DOCUMENT RESUME 



ED 451 223 



TM 032 451 



AUTHOR 

TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Lessik, Alan; Michener, Victoria 

Measuring Institutional Capacity. Recent Practices in 
Monitoring and Evaluation Tips, Number 15. 

Agency for International Development (IDCA) , Washington, DC. 
Center for Development Information and Evaluation. 

TIPS- 15 
2000 - 00-00 
27p . 

Guides - Non-Classroom (055) 

MF01/PC02 Plus Postage. 

Economic Development; *Evaluation Methods; *Institutional 
Characteristics; *Measurement Techniques; Organizational 
Development; *Program Evaluation; * Public Agencies 
* Agency for International Development; Capacity Building; 
*Monitoring 



ABSTRACT 



This document provides information for managers in the U. S. 
Agency for International Development on measuring institutional capacity, 
including some tools that measure the capacity of an entire organization as 
well as others that look at individual components or functions of an 
organization. The paper defines and discusses capacity assessment in general 
and presents several approaches for measuring capacity. The sections are: (1) 

"Background: Institutional Capacity Building and USAID"; (2) "How To Measure 
Institutional Capacity"; (3) "Measurement Issues"; (4) "Institutional 
Assessment Tools"; (5) "Measuring Individual Organizational Components"; (6) 
"Developing Indicators"; and (7) "Practical Tips for a Busy USAID Manager." 
The section on measurement issues is likely to be of interest to those 
interested in educational measurement. It focuses on comparability across 
organizations and over time, data collection, objectivity, quantification, 
internal versus external assessments, and practicality, all issues of 
interest to educational assessment. Other aspects of institutional assessment 
discussed would also have relevance for the educational practitioner. 
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MEASURING INSTITUTIONAL CAPACITY 



Introduction 



This PME Tips gives USAID managers information on measuring institutional capacity,* including some tools 
that measure the capacity of an entire organization as well as others that look at individual components or 
functions of an organization. The discussion concentrates on the internal capacities of individual organizations, 
rather than on the entire institutional context in which organizations function. ThisJz/w is not about how to 
actually strengthen” an institution, nor is it about how to assess the eventual impact of an organization’s 
work. Rather, it is limited to a specific topic: how to measure an institution’s capacities. It addresses the 
following questions: Which measurement approaches are most useful for particular types of capacity building? 
What are the strengths and limitations of each approach with regard to internal bias, quantification, or compa- 
rability over time or across organizations? How will the data be collected and how participatory can and should 
the measurement process be? Measuring institutional capacity might be one important aspect of a broader 
program in institutional strengthening; it may help managers make strategic, operational, or funding decisions; 
or it may help explain institutional strengthening activities and related performance. Whatever the reason for 
assessing institutional capacity, this Tips presents managers with several tools for identifying institutional 
strengths and weaknesses. 

The paper will define and discuss capacity assessment in general and present several approaches for measuring 
institutional capacity. We assess the measurement features of each approach to helpUSAID managers select the 
tool that best fits their diverse management and reporting needs. The paper is organized as follows: 

1. Background: Institutional Capacity Building and USAID 

2. How to Measure Institutional Capacity 



‘The terms institutional capacity, institution building, institutional strengthening, organizational capacity, and organizational develop- 
merit all address the same set of issues and are used here interchangeably. 



Recent Practices in Monitoring and Evaluation Tips explores a range of issues ' 
and discusses current experience and practices withinusAiD to help staff manage 
for results. Each paper covers a monitoring and evaluation topic in depth, comple- 
menting the shorter how-to guides that make up the bulk of the Tips series. Tips: 
contains no new Agency policy or essential procedures. Rather, the Tips series 
provides guidelines, advice, and suggestions to usaid managers on how to effec- 
tively plan and conduct performance monitoring and evaluation activities. 
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3. Measurement Issues 

4 . Institutional Assessment Tools 

5. Measuring Individual Organizational 
Components 

6. Developing Indicators 

7. Practical Tips for a Busy USAID Manager 

1. Background: Institutional Capacity 
Building and USAID 

USAID operating units must work closely with partner 
and customer organizations to meet program objec- 
tives across all Agency goal areas, among them 
economic growth and agricultural development, 
democracy and governance, and health. In the course 
of planning, implementing, and measuring their 
programs, USAID managers often find that a partner o: 
customer organization’s lack of capacity stands in the 
way of achieving results. Increasing the capacity of 
partner and customer organizations helps them carry 
out their mandate effectively and function more 
efficiently. Strong organizations are more able to 
accomplish their mission and provide for their own 
needs in the long run. 

USAID operating units build capacity with a broad 
spectrum of partner and customer organizations. 
These include but are not limited to 

■ American private voluntary organizations (PV Os) 



■ Local and international nongovernmental organi- 
zations (NGOs) and other civil society organiza- 
tions (CSOs) 

■ Community-based membership cooperatives, such 
as a water users’ group 

■ Networks and associations of organizations 

■ Political parties 

■ Government entities (ministries, departments, 
agencies, subunits, policy analysis units, health 
clinics, schools) 

■ Private sector organizations (financial institu- 
tions, companies, small businesses and other for- 
profit organizations) 

■ Regional institutions 

The Agency uses a variety of techniques to build 
organizational capacity. The most common involve 
providing technical assistance, advisory services, and 
long-term consultants to organizations, to help them 
build the skills and experience necessary to contribute 
successfully to sustainable development. Other 
techniques include providing direct inputs, such as 
financial, human, and technological resources. 

Finally, USAID helps establish mentoring relationships; 
provides opportunities for formal study in-country, in 
the United States or in third countries; and it sets up 
internships or apprenticeships with other organiza- 
tions. The goal of strengthening an institution is 
usually to improve the organization’s overall perfor- 
mance and viability by improving administrative and 



Capacity-building programs are intended to strengthen an organization’s ability to provide 
quality and effective services, while being .viable as an institution. This means supporting an 
organization to be programmatically sustainable (providing needed and effective services), 
as well as organizationally sustainable (with strong leadership and having necessary systems 
and procedures to manage by), while ensuring that it has sufficient resources (human, 
financial, and material) that are utilized well. Finally, this support must help the organization 
understand the external environment (political, economic, and social) it operates in, and to 
develop a. relationship with it that is sufficiently stable and predictable.. ... 

—New TransCentury Foundation, 1996, p.1 of vol. 3. 
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management functions, increasing the effectiveness of 
service provision, enhancing the organization’s 
structure and culture, and furthering its sustainability. 
Institutional strengthening programs may address one 
or more of these components. 

In most cases, USAID managers are concerned with 
institutional strengthening because they are interested 
in the eventual program-level results (and the 
sustainability of these results) that these stronger 
organizations can help achieve. While recognizing the 
need to address eventual results, this Tips looks 
primarily at ways to measure institutional capacity. 
Understanding and measuring institutional capacity 
are critical and often more complex than measuring 
the services and products an organization delivers. 

Measuring organizational capacity is important 
because it both guides USAID interventions and allows 
managers to demonstrate and report on progress. The 
data that emerge from measuring institutional capac- 
ity are commonly used in a number of valuable ways. 
These data establish baselines and provide the basis 
for setting targets for improvements. They help 
explain where or why something is going wrong; they 
identify changes to specific program interventions and 
activities that address areas of poor performance; they 
inform managers of the impact of an intervention or 
the effectiveness of an intervention strategy; and they 
identify lessons learned. They are also useful for 
reporting to Washington and to partners. 

It is important to note the difference between assess- 
ing capacity for contracting and grant-making deci- 
sions versus for a “capacity building” relationship 
with partner/customer organizations. A USAID man- 
ager may want to assess the capacity of an organiza- 
tion to help make decisions about awarding grants or 
holding grantees accountable for results. In this case, 
the assessment is more of an external oversight/audit 
of an organization hired to carry out Agency pro- 
grams. Or, the manager may have a programmatic 
commitment to strengthen the abilities of customer 
and partner organizations. Different tools and meth- 
ods are available for both situations. This paper deals 
primarily with programs that fit the latter description. 



Within USAID, the Office of Private and Voluntary 
Cooperation (PVC) in the Bureau for Humanitarian 
Response has taken the lead on building the capacity 
of nongovernmental organization (NGO) and private 
voluntary organization (PVO) partners. PVC has 
defined strategic objectives and intermediate results 
aimed specifically at improving the internal capacity 
of U.S. PVOs. PVC has studied different approaches to 
institutional capacity building and has begun to 
develop a comprehensive capacity assessment tool 
called discussion-oriented organizational self-assess- 
ment, described in example 1 in this paper. In addition 
to DOSA, PVC has developed several indicators for 
measuring institutional capacity development. 

PVC specifically targets NGOs and PVOs and is 
particularly concerned with enhancing partnerships. 
USAID missions, by contrast, work with a broader 
range of organizations on activities aimed at increas- 
ing institutional capacity. Such mission programs 
usually view institutional capacity as a means to 
achieve higher level program results, rather than as an 
end in itself. 

2. How to Measure Institutional Capacity 

An organization can be thought of as a system of 
related components that work together to achieve an 
agreed-upon mission. The following list of organiza- 
tional components is not all-inclusive, nor does it 
apply universally to all organizations. Rather, the 
components are representative of most organizations 
involved in development work and will vary according 
to the type of organization and the context in which it 
functions. 

Administrative and Support Functions 

■ Administrative procedures and management 
systems 

■ Financial management (budgeting, accounting, 
fundraising, sustainability) 

■ Human resource management (staff recruitment, 
placement, support) 



0 

ERIC 



5 




■ Management of other resources (information, 
equipment, infrastructure) 

Technical/Program Functions 

■ Service delivery system 

■ Program planning 

■ Program monitoring and evaluation 

■ Use and management of technical knowledge and 
skills 

Structure and Culture 

■ Organizational identity and culture 

■ Vision and purpose 

■ Leadership capacity and style 

■ Organizational values 

■ Governance approach 

■ External relations 
Resources 

■ Human 

■ Financial 

■ Other 

When selecting a measurement instrument, one should 
first clearly identify what needs to be measured. The 
intervention’s objective may be to strengthen the 
entire organization or only a specific function or 
component of the organization. The most useful 
measurement will capture only the information 
relevant to the intervention. 

In the field, practitioners can measure at several 
different levels. One approach is to measure the actual 
services or products the organization provides. A 
second method is to look at how well the entire 
organization functions by examining all of the compo- 



nents listed above. Yet another technique involves 
examining only selected components of the organiza- 
tion. 

In the first method, USAID program managers measure 
an organization’s outputs rather than its internal 
processes. In these cases, capacity building is clearly 
a means to an end. Operating units should and do 
track organizational results (quantity and quality of 
services provided). However, managers of institu- 
tional strengthening programs should also strengthen 
the organization’s processes so the organization can 
not only better achieve results, but do so in a sustain- 
able fashion. The second method is most useful when 
USAID is addressing the overall strengthening of an 
organization. It involves measuring the process that 
leads to better results — assessing the capacity of and 
tracking changes in all the components that make up 
an organization. The third approach, tracking only 
particular organizational components, is used when an 
organization is being strengthened in one or a few 
areas, such as financial management, leadership 
development, or program planning. 

3. Measurement Issues 

This Tips presents capacity-assessment tools and 
other measurement approaches that, while similar in 
some ways, vary in both their emphasis and their 
method for evaluating an organization’s capacity. 

Some use scoring systems and others don’t; some use 
questionnaires while others employ focus groups; 
some use external evaluators, and others use self- 
assessments; some emphasize problem solving, while 
others concentrate on appreciating organizational 
strengths. Some tools can be used to measure the 
same standard across many organizations, while 
others are organization specific. Many of the tools are 
designed so that the measurement process is just as 
important as, if not more important than, the resulting 
information. They may involve group discussions, 
workshops, or exercises, and may explicitly attempt to 
be participatory. Such tools try to create a learning 
opportunity for the organization’s members, so that 
the assessment itself becomes an integral part of the 
capacity-building effort. 

Because of each user’s different needs, it would be 
difficult to use this Tips as a screen to predetermine 




the best capacity-assessment tool for each situation. 
Rather, managers are encouraged to adopt the ap- 
proaches most appropriate to their program and to 
adapt the tools best suited for local needs. To assist 
managers in identifying the most useful tools and 
approaches, we consider the following issues for each 
of the tools presented: 

■ Type of organization measured. Many of the 
instruments developed to measure institutional 
capacity are designed specifically for measuring 
NGOs and PVOs. Most of these can be adapted 
easily for use with other types of organizations, 
including government entities. 

■ Comparability across organizations. To mea- 
sure multiple organizations, to compare them with 
each other, or to aggregate the results of activities 
aimed at strengthening more than one organiza- 
tion, the tool used should measure the same 
capacity areas for all the organizations and use 
the same scoring criteria and measurement 
processes. Note, however, that a standard tool, 
applied to diverse organizations, is less able to 
respond to specific organizational or environmen- 
tal circumstances. This is less of a problem if a 
group of organizations, using the same standard 
tool, has designed its diagnostic instrument 
together (see the following discussion of PROSE). 

■ Comparability over time. In many cases, the 
value of measuring institutional capacity lies in 
the ability to track changes in one organization 
overtime. That requires consistency in method 
and approach. A measurement instrument, once 
selected and adapted to the needs of a particular 
organization, must be applied the same way each 
time it is used. Otherwise, any shifts that are 
noted may reflect a change in the measurement 
technique rather than an actual change in the 
organization. 

■ Data collection. Data can be collected in a 
variety of ways: questionnaires, focus groups, 
interviews, document searches, and observation, 
to name only some. Some methods are hands-on 
and highly participatory, involving a wide range 
of customers, partners, and stakeholders, while 
others are more exclusive, relying on the opinion 



of one or two specialists. In most cases, it is best 
to use more than one data collection method. 

■ Objectivity. By their naftire, measures of institu- 
tional capacity are subjective. They rely heavily 
on individual perception, judgment, and interpre- 
tation. Some tools are better than others at 
limiting this subjectivity. For instance, they 
balance perceptions with more empirical observa- 
tions, or they clearly define the capacity area 
being measured and the criteria against which it is 
being judged. Nevertheless, users of these tools 
should be aware of the limitations to the findings. 

■ Quantification. Using numbers to represent 
capacity can be helpful when they are recognized 
as relative and not absolute measures. Many tools 
for measuring institutional capacity rely on 
ordinal scales. Ordinal scales are scales in which 
values can be ranked from high to low or more to 
less in relation to each other. They are useful in 
ordering by rank along a continuum, but they can 
also be misleading. Despite the use of scoring 
criteria and guidelines, one person’s “3” may be 
someone else’s “4.” In addition, ordinal scales do 
not indicate how far apart one score is from 
another. (For example, is the distance between 
“agree” and “strongly agree” the same as the 
distance between “disagree” and “strongly 
disagree”?) Qualitative descriptions of an 
organization’s capacity level are a good comple- 
ment to ordinal scales. 

■ Internal versus external assessments. Some tools 
require the use of external facilitators or assessors; 
others offer a process that the organization itself can 
follow. Both methods can produce useful data, and 
neither is automatically better than the other. 

Internal assessments can facilitate increased man- 
agement use and better understanding of an 
assessment’ s findings, since the members of the 
organization themselves are carrying out the 
assessment. By contrast, the risk of bias and 
subjectivity is higher in internal assessments. 
External assessments may be more objective. They 
are less likely to introduce internal bias and can 
make use of external expertise. The downside is that 
external assessors may be less likely to uncover 
what is really going on inside an organization. 
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■ Practicality. The best measurement systems are 
designed to be as simple as possible — not too time 
consuming, not unreasonably costly, yet able to 
provide managers with good information often 
enough to meet their management needs. Manag- 
ers should take practicality into account when 
selecting a measurement tool. They should 
consider the level of effort and resources required 
to develop the instrument and collect and analyze 
the data, and think about how often and at what 
point during the management cycle the data will 
be available to managers. 

4. Institutional Assesment Tools 

This section describes capacity measurement tools 
that USAID and other development organizations use. 
You can find complete references and Web sites in the 
resources section at the end of the paper. For each 
tool, we follow the same format. 

■ Background of the methodology/tool 

■ Process (how the methodology /tool is used in the 
field) 



■ An example of what the methodology/tool looks 
like 

Participatory, Results-Oriented 
Self-Evaluation 

Background 

The participatory, results-oriented self-evaluation 
(PROSE) method was developed by Evan Bloom of 
Pact and Beryl Levinger of the Education Develop- 
ment Center. It has the dual purpose of both assessing 
and enhancing organizational capacities. The PROSE 
method produces an assessment tool customized to the 
organizations being measured. It is designed to 
compare capacities across a set of peer organizations, 
called a cohort group, which allows for benchmarking 
and networking among the organizations. PROSE tools 
measure and profile organizational capacities and 
assess, over time, how strengthening activities affect 
organizational capacity. In addition, through a facili- 
tated workshop, PROSE tools are designed to allow 
organizations to build staff capacity; create consensus 
around future organizational capacity-building 
activities; and select, implement, and track organiza- 
tional change and development strategies. 



Product (the types of outputs expected) 

Assessment (a discussion of the uses and relative 
strengths of each methodology/tool) 



One example of an instrument developed using the 
PROSE method is the discussion-oriented organiza- 
tional self-assessment. DOSA was developed in 1997 
for the Office of Private and Voluntary Cooperation 
and was designed specifically for a cohort of USAID 
PVO grantees. 



<4^ Participatory, Results-Oriented Self-Evaluation 

Type of Organization Measured ji. 

NGOs/PVOs; adaptable to othertypes of organizations A--' 

Features' 

■ Cross-organizational comparisons can be made i 

■ Measures change in one organization or a cohort of organizations over time 

■ • Measures well-defined capacity areas- against well-defined criteria :??? v 

■ Assessment based primarily upon perceived capacities > 

■ Produces numeric score on capacity areas -:A; : V.\ 

■ Assessment should be done with the help of an outside facilitator or trained insider - 

■ Data collected through group discussion and individual questionnaires given to a cross-section of 
the organization’s staff 
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Process 

Developers of the PROSE method recommend that 
organizations participate inDOSA or develop a custom- 
ized DOSA-like tool to better fit their organization’s 
specific circumstances. The general PROSE process for 
developing such a tool is as' follows: After a cohort 
group of organizations is defined, the organizations meet 
in a workshop setting to design the assessment tool. With 
the help of a facilitator, they begin by pointing to the 
critical organizational capacities they want to measure 
and enhance. The cohort group then develops two sets of 
questions: discussion questions and individual question- 
naire items. The discussion questions are designed to get 
the group thinking about key issues. Further, these 
structured discussion questions minimize bias by 
pointing assessment team members toward a common 
set of events, policies, or conditions. The questionnaire 
items then capture group members’ assessments of those 
issues on an ordinal scale. During the workshop, both 
sets of questions are revised until the cohort group is 
satisfied. Near the end of the process, tools or standards 
from similar organizations can be introduced to check 
the cohort group’s work against an external example. If 
the tool is expected to compare several organizations 
within the same cohort group, the tool must be imple- 
mented by facilitators trained to administer it effectively 
and consistently across the organizations. 

Once the instrument is designed, it is applied to each 
of the organizations in the cohort. In the case of 
E)OSA, the facilitator leads a team of the organization’s 
members through a series of group discussions 
interspersed with individual responses to 1 00 ques- 
tionnaire items. The team meets for four to six hours 
and should represent a cross-functional, cross- 
hierarchical sample from the organization. Partici- 
pants respond anonymously to a questionnaire, 
selecting the best response to statements about the 
organization’s practices (l=strongly disagree, 
2=disagree, 3=neutral, 4=agree, 5=strongly agree) 
in six capacity areas: 

■ External Relations 

(constituency development, fund-raising 
and communications) 

■ Financial Resource Management 

(budgeting, forecasting, and cash management) 



■ Human Resource Management 

(staff training, supervision, and personnel 
practices) 

■ Organizational Learning 

(teamwork and information sharing) 

■ Strategic Management 

(planning, governance, mission, and partnering) 

■ Service Delivery 

(field-based program practices and sustainability 
issues) 

Although the analysis is statistically complex, ques- 
tionnaires can be scored and graphics produced using 
instructions provided on the DOSA Web site. In the 
case of DOSA, the DOSA team in Washington processes 
the results and posts them on the Internet. The assess- 
ment tool can be readministered annually to monitor 
organizational changes. 

Product 

PROSE instruments produce two types of scores and 
accompanying graphics. The first is a capacity score, 
which indicates how an organization perceives its 
strengths and weaknesses in each of the capacity and 
subcapacity areas. The second is a consensus score, 
which shows the degree to which the assessment team 
members agree on their evaluation of the 
organization’s capacity. 

Assessment 

Unless the existing DOSA questions are used, develop- 
ing a PROSE instrument from scratch can be time 
consuming and generally requires facilitators to guide 
the process of developing and using the instrument. 
PROSE, like most other such instruments, is based on 
perceived capacities and does not currently include a 
method for measuring externally observable perfor- 
mance in various capacity areas (although this is 
under consideration). It is unique among the instru- 
ments in this paper in its use of a consensus score. 

The consensus score acts as a check on the perceived 
capacities reported by individual organizational 
members. It also helps identify capacity areas that all 
members agree need immediate attention. 
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Example 1. Excerpt From DOSA, a PROSE Tool 

The DOSA questionnaire can be found in annex 1,® and more detailed information and instructs can 
be found on the USAID Web site. 

The following is a brief example drawn from the Human Resource Management section of thfe DOSA 
questionnaire: 


Discussion Questions 

a. When was our most recent staff training? 

b. How often over the last 12 months have we held staff training events? 






Questionnaire items for individual response 






Strongly Disagree Neutral 
disagree 


Agree 


Strongly 

agree 


1. We routinely offer staff training. 12 3 


4 


5 


Discussion Questions 

a. What are three primary, ongoing functions (e.g., monitoring and evaluation, proposal writing, resource 
mobilization) that we carry out to achieve our mission? 

b. To what extent does staff, as a group, have the requisite skills to carry out these functions? 

c. To what extent is the number of employees carrying out these functions commensurate with work 
demands? 


Questionnaire items for individual response 






Strongly Disagree Neutral 
disagree 


Agree 


Strongly 

agree 


2. We have the appropriate staff skills to achieve 12 3 

our mission 


4 


5 


3. We have the appropriate staff numbers to 1 2 3 

achieve our mission 


4 


5 



-The annexes for this paper are available separately and can be obtained through the USAID Internet, 
“Managing for Results’ page at http://www.dec.Org/usaid_eval/default.htm#004. 



Because the cohort organizations develop the specifics 
of the instrument together and share a common 
understanding and application of the approach, prose 
is relatively good at comparing organizations with 
each other or rolling up results to report on a group of 
organizations together. However, the discussions 
could influence the scoring if facilitators are not 
consistent in their administration of the tool. 

Institutional Development Framework 

Background 

The institutional development framework (EDF) is a 
tool kit developed by Mark Renzi of Management 
Systems International. It has been used in USAID/ 
Namibia’s Living in a Finite Environment project as 
well as several other USAID programs. Designed 



specifically to help nonprofit organizations improve 
efficiency and become more effective, theiDF is best 
suited for the assessment of a single organization, 
rather than a cohort group (as opposed to PROSE). The 
kit contains three tools (Institutional Development 
Framework, Institutional Development Profile, and 
Institutional Development Calculation Sheet), which 
help an organization determine where it stands on a 
variety of organizational components, identify priority 
areas of improvement, set targets, and measure 
progress over time. While it can be adapted for any 
organization, the IDF was originally formulated for 
environmental NGOs. 

Process 

An organization can use the EDF tools either with or 
without the help of a facilitator. The IDF identifies five 
organizational capacity areas, called resource charac- 
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Institutional Development Framework 



Type of Organization Measured 

ngos/pvos; adaptable to other types of organizations 
Features ' ' • ."'.-’v' 

■ Can be used, with limitations, to compare across organizations 



■ Measures change in the same organization overtime 

■ Measures well-defined capacity areas against well-defined criteria 
Assessment based primarily upon perceived capacities £ 

■ ' Produces numeric score on capacity areas 




■ Produces qualitative description of an organization’s capacity in terms of developmental stages 

■ Assessment can be done internally or wittvhelp of an outside facilitator :y.r •• 

■ Data collected through group discussion with as many staff as feasible 



teristics. Each capacity area is further broken down 
into key components, including 

■ Oversight/Vision 

(board, mission, autonomy) 

■ Management Resources 

(leadership style, participatory managment, 
management systems, planning, community 
participation, monitoring, evaluation) 

■ Human Resources 

(staff skills, staff development, 
organizational diversity) 

■ Financial Resources 

(financial management, financial 
vulnerability, financial solvency) 

■ External Resources 

(public relations, ability to work with local 
communities, ability to work with government 
bodies, ability to work with other NGOs) 

Each key component within a capacity area is rated at 
one of four stages along an organizational develop- 
ment continuum (1= start up, 2= development, 3= 
expansion/consolidation, and 4= sustainability), idf 
offers criteria describing each stage of development 
for each of the key components (see example 2 
below). 



Different processes can be used depending on the 
organization’s size and the desired outcome. Small 
organizations usually involve as many staff as pos- 
sible; larger organizations may work in small groups 
or use a few key informants. Members of the organi- 
zation can modify the Institutional Development 
Framework to fit their organization. Nonapplicable 
areas can be ignored and new areas can be added, 
although the creator of the tool warns against com- 
pletely rewriting the criteria. Through discussion, the 
participating members then use the criteria to deter- 
mine where along the development continuum their 
organization is situated for each component. The 
resulting graphic, the Institutional Development 
Profile (IDP), uses bars or “x”s to show where the 
organization ranks on each key component. Through a 
facilitated meeting or group discussion, organization 
members then determine which areas of organizational 
capacity are most important to the organization and 
which need priority attention for improvement. Using 
the IDP, they can visually mark their targets for the 
future. 

The IDF also provides numeric ratings. Each key 
component can be rated on a scale of 1 to 4, and all 
components can be averaged together to provide a 
summary score for each capacity area. This allows 
numeric targets to be set and monitored. The Institu- 
tional Development Calculation Sheet is a simple 
table that permits the organization to track progress 
over time by recording the score of each component 
along the development continuum. 
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. Example 2. Excerpt From the IDF Tool 

- The following is an excerpt from the Financial Management section of the Institutional Development 
Framework. The entire framework appears in annex 2. 


Resource 


Key 


Criteria for Each Proaressive Stage (the Development Continuum^ 


Characteristic 


Component 


Startup 

1 


Development 

2 


Expansion/ 

Consolidation 

3 


Sustainability 

4 


Financial 

Management 


Budget as 

Management 

Tools 


Budgets are 
not used as 
management 
tools. 


Budgets are 
developed for 
project activi- 
ties, but are 
often over- or 
underspent by 
more than 
20%. 


Total expendi- 
ture is usually 
within 20% of 
budget, but 
actual activity 
often diverges 
from budget 
predictions. 


Budgets are 
integral part of 
project manage- 
ment and are 
adjusted as 
project implemen- 
tation warrants. 




Cash 

Controls 


No clear 
procedures 
exist for 
handling 
payables and 
receivables. 


Financial 
controls exist 
but lack a 
systematic 
office proce- 
dure. 


Improved 
financial control 
systems exist. 


Excellent cash 
controls for 
payables and 
receivables and 
established 
budget proce- 
dures. 




Financial 

Security 


Financing 
comes from 
only one 
source. 


Financing 
comes from 
multiple 
sources, but 
90% or more 
from one 
source. 


No single 
source of 
funding pro- 
vides more 
than 60% of 
funding. 


No single source 
provides more 
than 40% of 
funding. 


*r* 


' '.$•’< jx ; v ' . ..<■< .. 

'Asii ■ 


si -.H i 






1 ' jf " 



Product Organizational Capacity Assesment Tool 



The IDF produces a graphic that shows the component 
parts of an organization and the organization’s ratings 
for each component at different points in time. It also 
provides a numeric score/rating of capacity in each 
key component and capacity area. 

Assessment 

The IDF is an example of a tool that not only helps 
assess and measure an organization’s capacity but 
also sets priorities for future change and improve- 
ments. Compared with some of the other tools, idf is 
relatively good at tracking one organization’s change 
over time because of the consistent criteria used for 
each progressive stage of development. It is probably 
not as well suited for making cross-organizational 
comparisons, because it allows for adjustment to fit 
the needs of each individual organization. 




Background 



Pact developed the organizational capacity assessment 
tool (OCAT*) in response to a need to examine the 
impact of NGO capacity-building activities. Like the 
Institutional Development Framework, ocat is better 
suited for measuring one organization over time. The 



OCAT differs substantially from the IDF in its data 
collection technique. It is designed to identify an 
organization’s relative strengths and weaknesses and 
provides the baseline information needed to develop 
strengthening interventions. It can also be used to 
monitor progress. The OCAT is well known; other 
development organizations have widely adapted it. 
Designed to be modified for each measurement 



*In 1996 Pact/Ethiopia designed the version of the OCAT 
presented here. 





Organizational Capacity Assesment Tool 

Type of Organization Measured 

NGOs/PVOs; adaptable to other types of organizations 

Features 

■ Gan be used, with limitations, to compare across organizations 

■ Measures change in the same organization over time 

■ Possible to measure well-defined capacity areas against well-defined criteria 

■ : Possible to balance perceptions with empirical observations 

■ Pnoduces numeric score on capacity areas : - 

■ Produces qualitative description of an organization’s capacity in terms of developmental stages 

■ Assessment can be done internally (with or without help of an outside facilitator) or by an external 

evaluator >• 

■ Data collected through group discussion, interviews, observations, documents, etc., by a diverse 
assessment team 



situation, the OCAT can also be standardized and used 
across organizations. 

Process 

The OCAT is intended to be a participatory self-assess- 
ment but may be modified to be an external evaluation. 
An assessment team, composed of organizational 
members (representing different functions of the organi- 
zation) plus some external helpers, modifies the OCAT 
assessment sheet to meet its needs (annex 3). The 
assessment sheet consists of a series of statements under 
seven capacity areas (with sub-elements). The assess- 
ment team then identifies sources of information, assigns 
tasks, and uses a variety of techniques (individual 
interviews, focus groups, among others) to collect the 
information they will later record on the assessment 
sheet. The assessment team assigns a score to each 
capacity area statement ( l=needs urgent attention and 
improvement; 2=needs attention; 3=needs improvement; 
4=needs improvement in limited aspects; but not major 
or urgent; 5=room for some improvement; 6=no need for 
immediate improvement). The assessment team would 
have to develop precise criteria for what rates as a “1 ” 
or a “2,” etc. 

The capacity areas and sub-elements are 



■ Management Practices 

(organizational structure, information 
management, administration procedures, 
personnel, planning, program development, 
program reporting) 

■ Human Resources 

(human resources development, staff roles, 
work organization, diversity issues, supervisory 
practices, salary and benefits) 

■ Financial Resources 

(accounting, budgeting, financial/inventory 
controls, financial reporting) 

■ Service Delivery 

(sectoral expertise, constituency, impact 
assessment) 

■ External Relations 

(constituency relations, inter-NGO collaboration, 
public relations, local resources, media) 

■ Sustainability 

(program/benefit sustainability, organizational 
sustainability, financial sustainability, resource 
base sustainability) 



■ Governance After gathering data, the assessment team meets to 

(board, mission/goal, constituency, leadership, reach a consensus on the rating of each element. With 
legal status) 
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Example 3. Excerpt From an Adaptation of the OCAT 

USAJD/Madagascar developed a capacity assessment tool based on the OCAT, but tailored it to its own 
need to measure 21 partner institutions implementing reproductive health programs, including the 
Ministry of Health. The mission tried to measure different types of organizations and compare them by 

creating a standardized instrument to use with all the organizations. 

Combining the OCAT results with additional information from facilitated discussions, the mission was 
able to summarize how different types of organizations perceived different aspects of their capacity and 
recommend future strengthening programs. •' 

Some of the difficulties that USAID/Madagascar encountered when using the tool included having to 
translate questions from French to Malagasy, possibly losing some of their meaning, finding that some 
. respondents were unable to answer some questions because they had no experience with the. part of 
the organization tawhich the questions referred; discovering that some respondents had difficulty 
separating the subject area of the questionnaire (family planning) from their work in other health areas, 
and having difficulty scheduling meetings because of the organizations’ heavy workload. Moreover, the 
mission noted thafcthe instrument is based on perceptions and is self-scored, with the resulting potential 
for bias. 3 {£•: 



Below is an excerpt from the “communications/extension to customers component of the OCAT used 
by USAID/Madagascar. The entire questionnaire is in annex 4. 




Classification Scale 

0 Nonexistent or out of order 

1 Requires urgent attention and upgrading 

2 Requires overall attention and upgrading 

3 Requires upgrading in certain areas, but neither major nor urgent 

4 Operating, but could benefit from certain improvements 

5 Operating well in all regards 

Communications/Extension to Customers 

a. The institution has in each clinic a staff trained and competent in coun- 1 2 3 4 5 

seling all customers. 

b. The institution is able to identify and develop key messages for exten- 1 2 3 4 5 

sion among potential customers, and it can produce or obtain materials 

for communicating such messages. 

c. A well-organized community extension is practiced by the clinic’s staff 1 2 3 4 5 

or other workers affiliated with the institution, whether they are salaried 

or volunteers. A system exists for supervising extension workers and 

monitoring their effective n ess. 

“USAID/Madagascar modeled its research on one element of the OCAT: the assessment sheet. Risks of bias decrease if 
self-assessment techniques are combined with other, more empirical methods, such as observations or document 
review 



the help of an OCAT rating sheet, averages can be 
calculated for each capacity area. These numeric 
scores indicate the relative need for improvement in 
each area. They also correspond to a more qualitative 
description of the organization’s developmental stage. 
Each capacity area can be characterized as nascent, 



emerging, expanding, or mature. OCAT provides a 
table (similar to the IDF), “NGO Organizational 
Development — Stages and Characteristics” that 
describes organizational capacities at each stage of 
development. 
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Product 

The OCAT provides numeric ratings for each capacity 
area. In addition, it gives organizations a description 
of their capacity areas in terms of progressive stages 
of organizational development. This information can 
be presented graphically as well as in narrative form. 

Assessment 

The OCAT identifies areas of organizational strength 
and weakness and tracks related changes from one 
measurement period to the next. 

The IDF and the OCAT are similar in several ways, but 
the processes differ. The OCAT uses an assessment 
team that conducts research before completing the 
assessment sheet. For the IDF, organization members 
meet and fill out the sheet (determine their capacities) 
without the intermediate data collection step (the 
OCAT, by design, relies on evidence to supplement 
perceptions when conducting an assessment, and the 
idf does not). The OCAT’s data-gathering step allows 
for systematic cross-checking of perceived capacities 
with actual or observable “facts.” It is more inductive, 
building up to the capacity description, while thelDF 
attempts to characterize the organization along the 
development continuum from the beginning. TheOCAT 
categorizes an organization’s capacity areas into one 
of four developmental stages. Unlike the IDF, which 
uses the stages as the criteria by which members rate 
their organization, the OCAT uses them as descriptors 
once the rating has been done. 



Dynamic Participatory 
Institutional Diagnosis 

Background 

The dynamic participatory institutional diagnosis (bPID) 
was developed by the Senegal PVO/NGO support project 
in conjunction with the New TransCentury Foundation 
and Yirawah International. It is a rapid and intensive 
facilitated assessment of the overall strengths and weak- 
nesses of an organization. This methodology explores 
member perceptions of an organization and the 
organization’s relationship with its environment.DPID 
is highly participatory; an organization assesses itself 
in the absence of external benchmarks or objectives to 
take full advantage of its specific context, such as cul- 
ture and attitudes. 

Process 

An outside facilitator conducts the DPID over 5 to 1 0 
days. It takes place during a series of working sessions 
in which the facilitator leads an organization’s mem- 
bers through several stages: discussion of the services; 
operations and results of the organization; exploration 
of the issues affecting the organization; and summari- 
zation of the “state of the organization.” During the 
discussions, members analyze the following features of 
the organization: 

■ Identity 

■ Mission 



Dynamic Participatory Institutional Diagnosis 

Type of Organization Measured ' r; 

ngos/pvos; adaptable to other types of organizations 

Features 

■ Difficult to compare across organizations 

■ Difficult to compare the same organization over time 

■ Capacity areas and criteria for measurement are loosely defined 

■ Assessment based primarily upon perceived capacities 

■ Produces qualitative description of an organization’s capacity 

■ Assessment done with the help of an outside facilitator 

■ Data collected through group discussion with the organization’s staff 
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Example 4. An Application of DPID 

' Since the DPID is such an individualized and flexible tool, every application will be different The dpid 
does not lend. itself easily to an example as do the other tools in this Tips. Below is an anecdote about 
one West African organization’s use of the DPID as reported by the Senegal DPIPVO/NGO support 
i project ' ■ f ' \ : : t ~ j " 



A Federation of Farmers’ Cooperatives with about 15,000 members in the Sahel was looking for a unique 
and efficient approach to redress some of the organization’s problems. The federation suffered from 
"internal strife and a tarnished reputation, impeding its ability to raise funds. Through DPID, the federation 
conducted a critical in-depth analysis of its operational and management systems, resulting in the 
adoption of “10 emergency measures” addressing leadership weaknesses, management systems, and 
operational procedures. Subsequently, the organization underwent internal restructuring, including an 
overhaul of financial and administrative systems. One'specific result of the DPID analysis was that 
federation members gained more influence over the operations of the federation. 




■ Means and Resources 

■ Environment 

■ Management 

■ Internal Operations 

■ Service Provided and Results 

They examine each element with reference to institu- 
tional behavior, human behavior, management, 
administration, know-how, philosophy and values, and 
sensitive points. 

Product 

A written description of the state of the organization 
can result from the working sessions. The analysis is 
qualitative without numeric scoring. 

Assessment 

Unlike the previously described tools, the DPID does 
not use ranking, scoring, or questionnaires, nor does it 
assess the organization along a continuum of develop- 
mental stages. Assessment is based purely on group 
reflection. The DPID requires a facilitator experienced 
in leading a group through this type of analysis. 

The DPID is open ended but somewhat systematic in 
covering a predefined set of organizational functions. 
Because of its flexibility, the DPID is organization- 
specific and should not be used to compare organiza- 



tions. Nor is it a rigorous means of monitoring an 
organization’s change overtime. Since the DPID does 
not use external standards to assess institutional 
capacities, it should not be used to track accountabil- 
ity. Collecting information from the DPID, as well as 
using it, should offer organizations a process to assess 
their needs, improve communications, and solve 
problems around a range of organizational issues at a 
given moment. 

Organizational Capacity Indicator 

Background 

From 1994 through 1997, the Christian Reformed 
World Relief Committee (CRWRC) conducted research 
on organizational capacity-building with the 
Weatherhead School of Management at Case Western 
Reserve University and more than 100 local NGOs 
around the world. The results of this research led them 
to replace their earlier system, the Skill Rating 
System, with an approach to capacity building and 
assessment based on “appreciative inquiry.” Apprecia- 
tive inquiry is a methodology that emphasizes an 
organization’s strengths and potential more than its 
problems. It highlights those qualities that give life to 
an organization and sustain its ongoing capacity. 
Rather than providing a standardized tool, the organi- 
zational capacity indicator assumes that capacity 
monitoring is unique to each organization and in the 
organization’s own self-interest. The organizational 
capacity indicator (OCI) builds ownership because 
each organization creates its own capacity assessment 
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Organizational Capacity Indicator 

Type of Organization Measured 

ngos/pvos; adaptable to other types of organizations 

Features : : , 

■ ; Difficult to comparably measure across organizations 

Measures change in the same organization over time .... 

■ Possible to measure well-defined capacity areas across well-defined criteria 

■ Assessment based primarily upon perceived capacities 

■ ; Produces numeric or pictorial score on capacity areas v 

■ Assessment done internally 

■ ■ Data collected through group discussion with organization’s staff 




tool. Capacity areas are self-defined and vary from 
organization to organization. 

Process 

Although organizations create their own tool under the 
OCI, they all follow a similar process in doing so. As 
they involve all partners and stakeholders as much as 
possible, the participants “appreciate” the 
organization’s history and culture. Together they 
explore peak experiences, best practices, and future 
hopes for the organization. Next, the participants 
identify the forces and factors that have made the 
organization’s positive experiences possible. These 
become the capacity areas that the organization tries 
to monitor and improve. 

Next, the participants develop a list of “provocative 
propositions” for each capacity area. These proposi- 
tions, visions of what each capacity area should 
ideally look like in the future, contribute to the overall 
objective that each organization will be able to 
measure itself against its own vision for the future, 
not some external standard. Each capacity area is 
defined by the most ambitious vision of what the 
organization can become in that area. Specific indica- 
tors or behaviors are then identified to show the 
capacity area in practice. Next, the organization 
designs a process for assessing itself and sharing 
experiences related to each capacity component. The 
organization should monitor itself by this process 
twice a year. The results of the assessment should be 



used to encourage future development, plans, and 
aspirations. 

Product 

Each time a different organization uses the method- 
ology, a different product specific to that organization 
is developed. Thus, each tool will contain a unique set 
of capacity areas, an evaluation process, and scoring 
methods. In general, the product comprises a written 
description of where the organization wants to be in 
each capacity area, a list of indicators that can be 
used to track progress toward the targeted level in a 
capacity area, and a scoring system. 

Assessment 

Like the DPID, the OCI is highly participatory and 
values internal standards and perceptions. Both tools 
explicitly reject the use of external standards. How- 
ever, the OCI does not designate organization capacity 
areas like the DPID does. The OCI is the only tool 
presented in this paper in which the capacity areas are 
entirely self defined. It is also unique in its emphasis 
on the positive, rather than on problems. Further, the 
OCI is more rigorous than the DPID, in that it asks 
each organization to set goals and develop indicators 
as part of the assessment process. It also calls for a 
scoring system to be developed, like the more formal 
tools (PROSE, IDF, OCAT). Because indicators and 
targets are developed for each capacity area, the tool 
allows for relatively consistent measurement over 
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Example 5. Excerpt From an OCI Tool 

The following is an excerpt of one section from the capacity assessment tool developed by CRWRC's 
partners in Asia, using the OCI method. (The entire tool can be found in annex 5.) It offers a menu of 
capacity areas and indicators from which an organization can choose and then modify for its own use. 
It identifies nine capacity areas, and undereach area is a “provocative proposition” or vision of where 
the organization wants to be in that area. It provides an extensive list of indicators for each capacity 
area, and it describes the process for developing and using the tool. Staff and partners meet regularly 
to determine their capacity on the chosen indicators. Capacity level can be indicated pictorially, for 
example by the stages of growth of a tree or degrees of happy faces. > 



Capacity Area 

A clear vision, mission, strategy, and set of shared values 

Proposition 

Our vision expresses our purpose for existing: our dreams, aspirations, and concerns for the 
poor. Our mission expresses how we reach our vision. Our strategy expresses the approach 
we use to accomplish our goals. The shared values that we hold create a common under- 
standing and inspire us to work together to achieve our goal. 



Selected Indicators 

■ Every person can state the mission and vision in his or her own words 

■ There is a yearly or a six-month plan, checked monthly 

■ Operations/activities are within the vision, mission, and goal of the organization 

■ Staff know why they do what they’re doing 

■ Every staff member has a clear workplan for meeting the strategy 

■ Regular meetings review and affirm the strategy 
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time. OCI is not designed to compare organizations 
with each other or to aggregate the capacity measures 
of a number of organizations; however, it has proven 
useful in allowing organizations to learn from each 
other and in helping outsiders assess and understand 
partner organizations. 

The Yes/No Checklist or ‘Scorecard’ 
Background 

A scorecard/checklist is a list of characteristics or 
events against which a yes/no score is assigned. These 
individual scores are aggregated and presented as an 
index. Checklists can effectively track processes, 
outputs, or more general characteristics of an organi- 
zation. In addition, they may be used to measure 
processes or outputs of an organization correlated to 
specific areas of capacity development. 

Scorecards/checklists can be used either to measure a 
single capacity component of an organization or 
several rolled together. Scorecards/checklists are 



designed to produce a quantitative score that can be 
used by itself or as a target (though a scorecard/ 
checklist without an aggregate score is also helpful). 

Process 

To construct a scorecard, follow these general steps: 
First, clarify what the overall phenomena to be 
measured are and identify the components that, when 
combined, cover the phenomenon fairly well. Next, 
develop a set of characteristics or indicators that 
together capture the relevant phenomena. If desired, 
and if evidence and analysis show that certain charac- 
teristics are truly more influential in achieving the 
overall result being addressed, define a weight to be 
assigned to each characteristic/indicator. Then rate the 
organization(s) on each characteristic using a well- 
defined data collection approach. The approach could 
range from interviewing organization members to 
reviewing organization documents, or it could consist 
of a combination of methods. Finally, if desired and 
appropriate, sum the score for the organization(s). 
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The Yes/No Checklist ‘Scorecard’ 

Type of Organization Measured 

All types of organizations 

Features 

■ Cross-organizational comparisons can be made 

■ Measures change in the same organization over time 

■ Measures well-defined capacity areas against well-defined criteria 

■ Possible to balance perceptions with empirical observations 

■ Produces numeric score on capacity areas 

■ Assessment can be done by an external evaluator or internally 
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Data collected through interviews, observation, documents, involving a limited numbOT.of ; staff ^ 



Product 

A scorecard/checklist results in a scored listing of 
important characteristics of an organization and can 
also be aggregated to get a summary score. 

Assessment 

A scorecard/checklist should be used when the 
characteristics to be scored are unambiguous. There is 
no room for “somewhat” or “yes, but . . with the 



scorecard technique. The wording of each characteris- 
tic should be clear and terms should be well defined. 
Because scorecards/checklists are usually based on 
observable facts, processes, and documents, they are 
more objective than most of the tools outlined in this 
Tips. This, in turn, makes them particularly useful for 
cross-organizational comparisons, or tracking organi- 
zations overtime; that is, they achieve better measure- 
ment consistency and comparability. Yet concentrating 
on observable facts can be limiting, if such facts are 
not complemented with descriptive and perception- 
based information. 



% Example 6. A Scorecard 

USAID/Mozambique developed the following scorecard to measure various aspects of institutional 
capacity in partner civil society organizations. The following example measures democratic gover- 
nance. ; 


Increased Democratic Governance Within Civil Society Organizations 


Characteristic 


Score 


Multiplied 
^ 


Weight 


Weighted 

score 


1. Leaders (board member or equivalent) of the CSO elected 
by secret ballot. No=0 pts. Yes=1 pt. 




X 


3 




2. General assembly meetings are adequately announced at 
least two weeks in advance to all members (1 pt.) and held at 
least twice a year (1 pt.). Otherwise=0 pt. 




X 


2 




3. Annual budget presented for member approval. No=0 pts. 
Yes=1 pt. 




X 


2 




4. Elected leaders separate from paid employees. No=0 pts. 
Yes=1 pt. 




X 


2 




5. Board meetings open to ordinary members (nonboard 
members). No=0 pts. Yes=1 pt. 




X 


1 




Total 
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Though a person outside the organization frequently 
completes the scorecard/checklist, self-assessment is 
also possible. Unlike other tools that require facilita- 
tors to conduct or interpret them, individuals who are 
not highly trained can also use scorecards. Further, 
since scorecards are usually tightly defined and 
specific, they are often a cheaper measurement tool. 

5. Measuring Individual 
Organizational Components 

In some cases, USAID is not trying to strengthen the 
whole organization, but rather specific parts of it that 
need special intervention. In many cases, the best way 
of measuring more specific organizational changes is 
to use portions of the instruments described. For 
instance, the IDF has a comparatively well-developed 
section on management resources (leadership style, 
participatory management, planning, monitoring and 
evaluation, and management systems). Similarly, the 
OCAT has some good sections on external relations 
and internal governance. 

Organizational development professionals also use 
other tools to measure specific capacity areas. Some 
drawbacks of these tools are that they require special- 
ized technical expertise and they can be costly to use 
on a regular basis. Other tools may require some 
initial training but can be much more easily institu- 
tionalized. Below we have identified some tools for 
measuring selected organizational components. (You 
will find complete reference information for these 
tools in the resources section of this Tips.) 

Structure and Culture 

The Preferred Organizational Structure instrument is 
designed to assess many aspects of organizational 
structure, such as formality of rules, communication 
lines, and decision-making. This tool requires organi- 
zational development skills, both to conduct the 
assessment and to interpret the results. 

Human Resources and Their Management 

Many personnel assessments exist, including the Job 
Description Index and the Job Diagnostic Survey, 
both of which measure different aspects of job 
satisfaction, skills, and task significance. However, 



skilled human resource practitioners must administer 
them. Other assessments, such as the Alexander Team 
Effectiveness Critique, have been used to examine the 
state and functioning of work teams and can easily be 
applied in the field. 

Service Delivery 

Often, a customer survey is one of the best ways to 
measure the efficiency and effectiveness of a service 
delivery system. A specific customer survey would 
need to be designed relative to each situation. 

Example 7 shows a sample customer service assess- 
ment. 

6. Developing Indicators 

Indicators permit managers to track and understand 
activity/program performance at both the operational 
(inputs, outputs, processes) and strategic (strategic 
objectives and intermediate results) levels. To manag- 
ers familiar with the development and use of indica- 
tors, it may seem straightforward to derive indicators 
from the instruments presented in the preceding pages. 
However, several critical points will ensure that the 
indicators developed within the context of these 
instruments are useful to managers. 

First, the development of indicators should be driven 
by the informational needs of managers, from both 
USAID and the given relevant organizations — to 
inform strategic and operational decisions and to 
assist in reporting and communicating to partners and 
other stakeholders. At times, there is a tendency to 
identify or design a data collection instrument without 
giving too much thought to exactly what information 
will be needed for management and reporting. In these 
situations, indicators tend to be developed on the basis 
of the data that have been collected, rather than on 
what managers need. More to the point, the develop- 
ment of indicators should follow a thorough assess- 
ment of informational needs and precede the identifi- 
cation of a data collection instrument. Managers 
should first determine their informational needs; from 
these needs, they should articulate and define indica- 
tors; and only then, with this information in hand, they 
would identify or develop an instrument to collect the 
required data. This means that, in most cases, indica- 
tors should not be derived, post facto , from a data 
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collection tool. Rather, the data collection tool should 
be designed with the given indicators in mind. 

Second, indicators should be developed for manage- 
ment decisions at all levels (input indicators, output 
indicators, process indicators, and outcome/results 
indicators). With USAID’s increased emphasis on 
results, managers sometimes may concentrate prima- 
rily on results- level indicators (for strategic objectives 



and intermediate results). While an emphasis on 
results is appropriate, particularly for USAID manag- 
ers, tracking operational-level information for the 
organizations supported through a given Agency 
program is critical if managers are to understand if, to 
what degree, and how the organizations are increasing 
their capacities. The instruments outlined in this paper 
can provide data for indicators defined at various 
management levels. 




Example 7. A Customer Service Assessment 









S USP 



1 . In the past 12 months, have you ever contacted a municipal office to complain about something such as 
poor city services or a rude city official, or any other reason? 

No Yes 



If YES: 

1 a. How many different problems or complaints did you contact the municipality about in the last 1 2 months? 
One Two Three to five More than five 



1b. Please describe briefly the nature of the complaint starting with the one you feel was most important. 

1 . 

2. 

3. 

2. Which department or officials did you contact initially regarding these complaints? 

Mayor’s office 

Council member 

Police 

Sanitation 

Public works 

Roads 

Housing 

Health 

Other 

2a. Were you generally satisfied with the city’s response? (IF DISSATISFIED, ASK: What were the major 
reasons for your dissatisfaction?) 

Response not yet completed 

Satisfied 

Dissatisfied, never responded or corrected condition 

Dissatisfied, poor quality, or incorrect response was provided 

Dissatisfied, took too long to complete response, had to keep pressuring for results, red tape, etc. 

Dissatisfied, personnel were discourteous, negative, etc. 

Dissatisfied, other 

3. Overall, are you satisfied with the usefulness, courtesy and effectiveness of the municipal department or 
official that you contacted? 

Definitely yes 

Generally yes 

Generally no (explain) 

Definitely no (explain) 



Survey adapted from Hatry, Blair, and others, 1 992. 
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Finally, indicators should meet the criteria outlined in 
USAID’s Automated Directives System and related 
pieces of Agency guidance such as CDlE’s Perfor- 
mance Monitoring and Evaluation Tips #6, “Selecting 
Performance Indicators,” and Tips #12, “Guidelines 
for Indicator and Data Quality.” That is, indicators 
should be direct, objective, practical, and adequate. 
Once an indicator has been decided upon, it is impor- 
tant to document the relevant technical details: a 
precise definition of the indicator; a detailed descrip- 
tion of the data source; and a thorough explanation of 
the data collection method. (Refer to Tips #7, “Pre- 
paring a Performance Monitoring Plan.”) 

Results-Level Indicators 

USAID managers spend substantial time and energy 
developing indicators for strategic objectives and 
intermediate results related to institutional capacity. 
The range of the Agency’s institutional strengthening 
programs is broad, as is the range of the indicators 
that track the programs’ results. Some results reflect 
multiple organizations and others relate to a single 
organization. Additionally, of those results that relate 
to multiple organizations, some may refer to organiza- 
tions from only one sector while others may capture 
organizations from a number of sectors. Results 
related to institutional strengthening also vary relative 
to the level of change they indicate — such as an 
increase in institutional capacity versus the eventual 
impact generated by such an increase — and with 



regard to whether they reflect strengthening of the 
whole organization(s) or just one or several elements. 

It is relatively easy to develop indicators for all types 
of results and to use the instruments outlined in this 
Tips to collect the necessary data. For example, when 
a result refers to strengthening a single organization, 
across all elements, an aggregate index or “score” of 
institutional strength may be an appropriate indicator 
(an instrument based on the IDF or the scorecard 
model might be used to collect such data). If a result 
refers to multiple organizations, it might be useful to 
frame an indicator in terms of the number or percent 
of the organizations that meet or exceed a given 
threshold score or development stage, on the basis of 
an aggregate index or the score of a single element for 
each organization. The key is to ensure that the 
indicator reflects the result and to then identify the 
most appropriate and useful measurement instrument. 

Below are examples of real indicators used by USAID 
missions in 1998 to report on strategic objectives and 
intermediate results in institutional capacity strength- 
ening: 

7. Practical Tips for a Busy 
USAID Manager 

This Tips introduces critical issues related to measur- 
ing institutional capacity. It presents a number of 
approaches that managers of development programs 
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k- Example 8. Selected Institutional Capacity Indicators From USAID Missions 


Indicator 


To Measure 


■ Number of institutions meeting at least 80% of 
their targeted improvements 


Institutions strengthened (entire organization) 


■ Amount of funds raised from non-USAID sources 

■ Number of organizations where USAID contribu- 
tion is less than 25% of revenues 

■ Number of organizations where at least five 
funding sources contribute at least 10% each 


Institutions more financially sustainable 


■ Percent of suspected polio cases investigated 
within 48 hours 


Organization’s service delivery systems 
strengthened 


■ Number of governmental units displaying 
improved practices, such as open and trans- 
parent financial systems, set organizational 
procedures, accountability, participatory 
decision-making, by-laws and elections 


Local government management capacities 
improved 
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and activities currently use in the field. In this section 
we summarize the preceding discussion by offering 
several quick tips that usaid managers should find 
useful as they design, modify, and implement their 
own approaches for measuring institutional capacity. 

1. Carefully review the informational needs of the 
relevant managers and the characteristics of the 
organization to be measured to facilitate develop- 
ment of indicators. Identify your information needs 
and develop indicators before you choose an 
instrument. 

2. To assist you in selecting an appropriate measure- 
ment tool, ask yourself the following questions as 
they pertain to your institutional capacity mea- 
surement situation. Equipped with the answers to 
these questions, you can scan the “features list” 
that describes every tool in this paper to identify 
which measurement approaches to explore further. 

a. Is the objective to measure the entire organiza- 
tion? Or is it to measure specific elements of the 
organization? If the latter, what are the specific 
capacity areas of functions to be measured? 

b. How will the information be used? To measure 
change in an organization over time? To com- 
pare organizations with each other? 

c. What is the purpose of the intervention? To 
strengthen an organization? To inform procure- 
ment decisions? To hold an organization ac- 
countable for achieving results or implementing 
reforms? 

d. What type of organizations are you measuring? 
Are there any particular measurement issues 
pertaining to this type of organization that must 
be considered? 

e. How participatory do you want the measurement 
process to be? 

f. Will organization members themselves or outsid- 
ers conduct the assessment? 

g. What product do you want the measurement tool 
to generate? 



h. Do you want the measurement process to be an 
institution-strengthening exercise in itself? 

i. Do you need an instrument that measures one 
organization? Several organizations against 
individual criteria? Or several organizations 
against standard criteria? 

3. If you are concerned about data reliability , apply 
measurement instruments consistently over time 
and across organizations to ensure data reliability. 
You can adapt and adjust tools as needed, but once 
you develop the instrument, use it consistently. 

4. When interpreting and drawing conclusions from 
collected data , remember the limits of the relevant 
measurement tool. Most methods for measuring 
institutional capacity are subjective, as they are 
based on the perceptions of those participating in 
the assessment, and involve some form of ordinal 
scaling/scoring. When reviewing data, managers 
should therefore zero in on the direction and 
general degree of change. Do not be overly 
concerned about small changes; avoid false preci- 
sion. 

5. Cost matters— and so does the frequency and 
timing of data collection. Data need to be available 
frequently enough, and at the right point in the 
program cycle, to inform operational and strategic 
management decisions. Additionally, the manage- 
ment benefits of data should exceed the costs 
associated with their collection. 

6. The process of measuring institutional capacity 
can contribute substantially to increasing an 
organization f s strength A number of measurement 
approaches are explicitly designed as learning 
opportunities for organizations; that is, to identify 
problems and suggest related solutions, to improve 
communication, or to facilitate a consensus around 
future priorities. 



This Tips was prepared for cdie by 
Alan Lessik and Victoria Michener of 
Management Systems International. 
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This Recent Practices in Monitoring and Evaluation Tips is intended to present some of the current 
practices USAID managers use to measure institutional capacity. The purpose is not only to share 
this information and experience but also to engender discussion on new and existing best prac- 

\ dices in measuring institutional change. If you have specific comments, questions or experiences 

best practices — that you would like to share with us, please contact John Haecker, PPC/cdie/pme 
■ 0haecker@uscdd.gov). .-m- 
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